{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ur8xi4C7S06n"
   },
   "outputs": [],
   "source": [
    "# Copyright 2021 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAPoU8Sm5E6e"
   },
   "source": [
    "<table align=\"left\">\n",
    "\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/notebook_template.ipynb\"\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/notebook_template.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Overview\n",
    "\n",
    "{TODO: Include a paragraph or two explaining what this example demonstrates, who should be interested in it, and what you need to know before you get started.}\n",
    "\n",
    "### Dataset\n",
    "\n",
    "{TODO: Include a paragraph with Dataset information and where to obtain it.} \n",
    "\n",
    "{TODO: Make sure the dataset is accessible to the public. **Googlers**: Add your dataset to the [public samples bucket](http://goto/cloudsamples#sample-storage-bucket) within gs://cloud-samples-data/ai-platform-unified, if it doesn't already exist there.}\n",
    "\n",
    "### Objective\n",
    "\n",
    "In this notebook, you will learn how to {TODO: Complete the sentence explaining briefly what you will learn from the notebook, such as\n",
    "training, hyperparameter tuning, or serving}:\n",
    "\n",
    "    * {TODO: Add high level bullets for the steps of what you will perform in the notebook}\n",
    "\n",
    "### Costs \n",
    "\n",
    "{TODO: Update the list of billable products that your tutorial uses.}\n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "\n",
    "* Vertex AI\n",
    "* Cloud Storage\n",
    "\n",
    "{TODO: Include links to pricing documentation for each product you listed above.}\n",
    "\n",
    "Learn about [Vertex AI\n",
    "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
    "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
    "Calculator](https://cloud.google.com/products/calculator/)\n",
    "to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ze4-nDLfK4pw"
   },
   "source": [
    "### Set up your local development environment\n",
    "\n",
    "**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
    "all the requirements to run this notebook. You can skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gCuSR8GkAgzl"
   },
   "source": [
    "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
    "You need the following:\n",
    "\n",
    "* The Google Cloud SDK\n",
    "* Git\n",
    "* Python 3\n",
    "* virtualenv\n",
    "* Jupyter notebook running in a virtual environment with Python 3\n",
    "\n",
    "The Google Cloud guide to [Setting up a Python development\n",
    "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
    "installation guide](https://jupyter.org/install) provide detailed instructions\n",
    "for meeting these requirements. The following steps provide a condensed set of\n",
    "instructions:\n",
    "\n",
    "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
    "\n",
    "1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
    "\n",
    "1. [Install\n",
    "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
    "   and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
    "\n",
    "1. To install Jupyter, run `pip3 install jupyter` on the\n",
    "command-line in a terminal shell.\n",
    "\n",
    "1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
    "\n",
    "1. Open this notebook in the Jupyter Notebook Dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Install additional packages\n",
    "\n",
    "Install additional package dependencies not installed in your notebook environment, such as {XGBoost, AdaNet, or TensorFlow Hub TODO: Replace with relevant packages for the tutorial}. Use the latest major GA version of each package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wyy5Lbnzg5fi"
   },
   "outputs": [],
   "source": [
    "! pip3 install {USER_FLAG} --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EzrelQZ22IZj"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lWEdiXsJg0XY"
   },
   "source": [
    "## Before you begin\n",
    "\n",
    "### Select a GPU runtime\n",
    "\n",
    "**Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select \"Runtime --> Change runtime type > GPU\"**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BF1j6f9HApxa"
   },
   "source": [
    "### Set up your Google Cloud project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
    "\n",
    "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
    "\n",
    "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). {TODO: Update the APIs needed for your tutorial. Edit the API names, and update the link to append the API IDs, separating each one with a comma. For example, container.googleapis.com,cloudbuild.googleapis.com}\n",
    "\n",
    "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
    "\n",
    "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\n",
    "\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "#### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oM1iC_MfAts1"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"\"\n",
    "\n",
    "# Get your Google Cloud project ID from gcloud\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJYoRfYng0XZ"
   },
   "source": [
    "Otherwise, set your project ID here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "riG_qUokg0XZ"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
    "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "06571eb4063b"
   },
   "source": [
    "#### Timestamp\n",
    "\n",
    "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "697568e92bd6"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dr--iN2kAylZ"
   },
   "source": [
    "### Authenticate your Google Cloud account\n",
    "\n",
    "**If you are using Google Cloud Notebooks**, your environment is already\n",
    "authenticated. Skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sBCra4QMA2wR"
   },
   "source": [
    "**If you are using Colab**, run the cell below and follow the instructions\n",
    "when prompted to authenticate your account via oAuth.\n",
    "\n",
    "**Otherwise**, follow these steps:\n",
    "\n",
    "1. In the Cloud Console, go to the [**Create service account key**\n",
    "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
    "\n",
    "2. Click **Create service account**.\n",
    "\n",
    "3. In the **Service account name** field, enter a name, and\n",
    "   click **Create**.\n",
    "\n",
    "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
    "into the filter box, and select\n",
    "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
    "\n",
    "5. Click *Create*. A JSON file that contains your key downloads to your\n",
    "local environment.\n",
    "\n",
    "6. Enter the path to your service account key as the\n",
    "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PyQmSRbKA8r-"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "# If you are running this notebook in Colab, run this cell and follow the\n",
    "# instructions to authenticate your GCP account. This provides access to your\n",
    "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
    "# requests.\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# If on Google Cloud Notebooks, then don't execute this code\n",
    "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    if \"google.colab\" in sys.modules:\n",
    "        from google.colab import auth as google_auth\n",
    "\n",
    "        google_auth.authenticate_user()\n",
    "\n",
    "    # If you are running this notebook locally, replace the string below with the\n",
    "    # path to your service account key and run this cell to authenticate your GCP\n",
    "    # account.\n",
    "    elif not os.getenv(\"IS_TESTING\"):\n",
    "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgPO1eR3CYjk"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "\n",
    "{TODO: Adjust wording in the first paragraph to fit your use case - explain how your tutorial uses the Cloud Storage bucket. The example below shows how Vertex AI uses the bucket for training.}\n",
    "\n",
    "When you submit a training job using the Cloud SDK, you upload a Python package\n",
    "containing your training code to a Cloud Storage bucket. Vertex AI runs\n",
    "the code from this package. In this tutorial, Vertex AI also saves the\n",
    "trained model that results from your job in the same bucket. Using this model artifact, you can then\n",
    "create Vertex AI model and endpoint resources in order to serve\n",
    "online predictions.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
    "Cloud Storage buckets.\n",
    "\n",
    "You may also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
    "available](https://cloud.google.com/vertex-ai/docs/general/locations#available_regions). You may\n",
    "not use a Multi-Regional Storage bucket for training with Vertex AI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MzGDU7TWdts_"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}\n",
    "REGION = \"[your-region]\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cf221059d072"
   },
   "outputs": [],
   "source": [
    "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
    "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [],
   "source": [
    "! gsutil mb -l $REGION $BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ucvCsknMCims"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vhOb7YnwClBb"
   },
   "outputs": [],
   "source": [
    "! gsutil ls -al $BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y9Uo3tifg1kx"
   },
   "source": [
    "{TODO: Put all your imports and installs up into a setup section.}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pRUOFELefqf1"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E6ppE7imft-y"
   },
   "source": [
    "## General style examples\n",
    "\n",
    "### Notebook heading\n",
    "\n",
    "- Include the collapsed license at the top (this uses Colab's \"Form\" mode to hide the cells).\n",
    "- Only include a single H1 title.\n",
    "- Include the button-bar immediately under the H1.\n",
    "- Check that the Colab and GitHub links at the top are correct.\n",
    "\n",
    "### Notebook sections\n",
    "\n",
    "- Use H2 (##) and H3 (###) titles for notebook section headings.\n",
    "- Use [sentence case to capitalize titles and headings](https://developers.google.com/style/capitalization#capitalization-in-titles-and-headings). (\"Train the model\" instead of \"Train the Model\")\n",
    "- Include a brief text explanation before any code cells.\n",
    "- Use short titles/headings: \"Download the data\", \"Build the model\", \"Train the model\".\n",
    "\n",
    "### Writing style\n",
    "\n",
    "- Use [present tense](https://developers.google.com/style/tense). (\"You receive a response\" instead of \"You will receive a response\")\n",
    "- Use [active voice](https://developers.google.com/style/voice). (\"The service processes the request\" instead of \"The request is processed by the service\")\n",
    "- Use [second person](https://developers.google.com/style/person) and an imperative style. \n",
    "    - Correct examples: \"Update the field\", \"You must update the field\"\n",
    "    - Incorrect examples: \"Let's update the field\", \"We'll update the field\", \"The user should update the field\"\n",
    "- **Googlers**: Please follow our [branding guidelines](http://goto/cloud-branding).\n",
    "\n",
    "### Code\n",
    "\n",
    "- Put all your installs and imports in a setup section.\n",
    "- Save the notebook with the Table of Contents open.\n",
    "- Write Python 3 compatible code.\n",
    "- Follow the [Google Python Style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md) and write readable code.\n",
    "- Keep cells small (max ~20 lines)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "euaAioez2WyE"
   },
   "source": [
    "## TensorFlow code style\n",
    "\n",
    "Use the highest level API that gets the job done (unless the goal is to demonstrate the low level API). For example, when using Tensorflow:\n",
    "\n",
    "- Use TF.keras.Sequential > keras functional api > keras model subclassing > ...\n",
    "\n",
    "- Use  model.fit > model.train_on_batch > manual GradientTapes.\n",
    "\n",
    "- Use eager-style code.\n",
    "\n",
    "- Use tensorflow_datasets and tf.data where possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "reBCSTKOg47l"
   },
   "source": [
    "### Notebook code style examples\n",
    "\n",
    " - Notebooks are for people. Write code optimized for clarity.\n",
    "\n",
    " - Demonstrate small parts before combining them into something more complex. Like below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MjJTYC86hPOZ"
   },
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model = tf.keras.Sequential(\n",
    "    [\n",
    "        tf.keras.layers.Dense(10, activation=\"relu\", input_shape=(None, 5)),\n",
    "        tf.keras.layers.Dense(3),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LWymd3KPhP_S"
   },
   "outputs": [],
   "source": [
    "# Run the model on a single batch of data, and inspect the output.\n",
    "result = model(tf.constant(np.random.randn(10, 5), dtype=tf.float32)).numpy()\n",
    "\n",
    "print(\"min:\", result.min())\n",
    "print(\"max:\", result.max())\n",
    "print(\"mean:\", result.mean())\n",
    "print(\"shape:\", result.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y-pQzaxnhR-0"
   },
   "outputs": [],
   "source": [
    "# Compile the model for training\n",
    "model.compile(\n",
    "    optimizer=tf.keras.optimizers.Adam(), loss=tf.keras.losses.categorical_crossentropy\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g0d8u7FmhUNf"
   },
   "source": [
    "- Keep examples quick. Use small datasets, or small slices of datasets. You don't need to train to convergence, train until it's obvious it's making progress.\n",
    "\n",
    "- For a large example, don't try to fit all the code in the notebook. Add python files to tensorflow examples, and in the notebook run: \n",
    "! pip3 install git+https://github.com/tensorflow/examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this tutorial:\n",
    "\n",
    "{TODO: Include commands to delete individual resources below}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sx_vKniMq9ZX"
   },
   "outputs": [],
   "source": [
    "# Delete endpoint resource\n",
    "! gcloud ai endpoints delete $ENDPOINT_NAME --quiet --region $REGION_NAME\n",
    "\n",
    "# Delete model resource\n",
    "! gcloud ai models delete $MODEL_NAME --quiet\n",
    "\n",
    "# Delete Cloud Storage objects that were created\n",
    "! gsutil -m rm -r $JOB_DIR"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "notebook_template.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
